FrBMedQA: the first French biomedical question answering dataset

FrBMedQA is the first French biomedical question answering dataset, containing 41k+ passage-question instances. It was automatically constructed in a cloze-style manner, from Wikipedia articles. To test validity and difficulty of we experimented with four statistical baseline models, bidirectional encoder representation transformers (BERT)-based model, two BERT-based language model. We also did human evaluation on subset set. All three tested models were not able to surpass best performing Human performance at 61.11% leading leaderboard more than 8% made available dataset code reproduce our results.